Apparatus and method for image coding and decoding

ABSTRACT

A computer-readable medium has data stored thereon for processing by an image decoding apparatus so that multimedia content is presented in an intended manner by a display. A data recording area contains data representing a converted video stream, at least another stream, and multimedia coding data for controlling display of the video stream and the at least another stream on a common display device. The converted video stream is generated by performing a predetermined conversion process on an original video stream. The predetermined conversion process is controlled by additional information such that a display mismatch between the converted video stream and the at least another stream is avoided. The additional information is based on the multimedia coding data.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. application Ser. No.09/872,147, filed Jun. 1, 2001, which claims priority from JapaneseApplication No. P2000-165298, filed Jun. 2, 2000, and JapaneseApplication No. P2001-001031, filed Jan. 9, 2001, the disclosures ofwhich are incorporated by reference herein.

BACKGROUND OF THE INVENTION

The present invention relates generally to an image coding apparatus andmethod, an image decoding apparatus and method, and a recording medium.More specifically, the present invention relates to an image codingapparatus and method, an image decoding apparatus and method, and arecording medium which are suitable for use in apparatus for re-encodingvideo streams and recording and reproducing the re-encoded videostreams.

Digital television broadcasts such as European DVB (Digital VideoBroadcast), American DTV (Digital Television) broadcast, and Japanese BS(Broadcast Satellite) digital broadcast use MPEG (Motion Picture ExpertGroup) 2 transport streams. A transport stream consists of continuoustransport packets, each packet carrying video data or audio data, forexample. The data length of one transport packet is 188 bytes.

Unlike analog television broadcasts, digital television broadcasts arecapable of providing services added with multimedia coding data. Inthese services, data such as video data, audio data, character graphicsdata, and still picture data, for example, are associated with eachother for transmission by the multimedia coding data. For the multimediacoding data, a coding method based on XML (Extensible Markup Language)is used in the Japanese BS digital broadcast, for example. The detailsof this method are disclosed in ARIB STD-B24 Data Coding AndTransmission Specification for Digital Broadcasting, for example.

Data such as video data, audio data, character graphics data, and stillpicture data are each packetized into a transport packet fortransmission.

FIGS. 1A and 1B show an example of synthesizing data to be transferredbetween the sending and receiving sides and a multimedia screen. Asshown in FIG. 1A, the sending side sends to the receiving side videodata, character graphics data for displaying buttons A through C, textdata for displaying “XYZABC . . . ,” and multimedia coding data forrelating these data to each other. The sending side generally denotes atelevision broadcast station, for example. However, herein it denotes atelevision broadcast station which includes a recording apparatus (therecording side) which receives and records data transmitted frombroadcast stations, as shown in the example illustrated in FIG. 1Aincluding the data which is output from this recording apparatus.

The multimedia coding data includes data which can synthesize on thereceiving side video data, character graphics data, and text data anddisplay the synthesized data. To be more specific, the multimedia codingdata includes the data associated with the display positions of thevideo, character graphics, and text which are displayed by thesize-associated data such as the multimedia plane (the display area ofimages on the television receiver, for example) size (plane_height andplane_width) and video display size (video_height and video_width),video data, character graphics data, and text data, as shown in FIG. 1B.

On the basis of the multimedia coding data, the receiving side processesthe video data, the character graphics data, and the text data todisplay a resultant image, as shown in FIG. 1B.

Through the screen on which the above-mentioned image is displayed, theuser can receive services such as displaying desired information in thevideo section by clicking button A corresponding to that information andobtaining, from the text data displayed in the bottom of the screen, theinformation associated with the matter displayed in the video section,for example.

If a television program carried by a transport stream transmitted from adigital television broadcast is recorded without change to a recordingmedium on the received side, the program can be recorded without itspicture and audio qualities being deteriorated at all. However, in orderto record as long a television program as possible to a recording mediumhaving a limited recording capacity by presupposing a certain degree ofpicture quality deterioration, the received video stream must be decodedand then encoded again to lower the bit rate of the transport stream.

For example, the re-encoding of the video stream of a television programattached with multimedia coding data to lower its bit rate for recordingmay be implemented by sub-sampling the image to change writing blocks.However, this approach presents a problem of causing a mismatch in therelationship between the video stream resulting from re-encoding and themultimedia coding data. The following describes an example of thismismatch with reference to FIGS. 2A and 2B.

In the example shown in FIG. 2A, the sending side (the recording side)converts the original video writing block to a smaller picture frame atthe time of re-encoding. Therefore, as shown in FIG. 2B, on thereceiving side (the reproducing side), changes occur in the videodisplay size and position, resulting in a display screen which isdifferent from the display screen intended by the sending side (thedisplay screen to be displayed on the basis of the data before beingre-encoded).

It is therefore desirable to provide an image coding apparatus andmethod, an image decoding apparatus and method, and a recording medium,wherein there is no mismatch of information caused in the relationshipbetween a video stream after re-encoding and other data is generated andrecorded on the recording side and reproduction on the reproducing side.For example, it is desirable to prevent a smaller picture frame fromoccurring on the reproducing side by referencing the informationgenerated on the recording side.

SUMMARY OF THE INVENTION

In carrying out the invention and according to a first aspect thereof,there is provided a first image coding apparatus comprising: inputtingmeans for inputting a multiplexed stream containing multimedia codingdata; separating means for separating a video stream from themultiplexed stream input to the separating means from the inputtingmeans; converting means for performing a predetermined convertingprocess on the video stream separated by the separating means;generating means for generating additional information indicating that amismatch will occur when the converted video stream is displayed on thebasis of the multimedia coding data; and outputting means for outputtingthe converted video stream, the multimedia coding data, and theadditional information.

The first image coding apparatus may further comprise: coding means forcoding the additional information generated by the generating means asdata separate from the multiplexed stream containing the converted videostream.

The first image coding apparatus may further comprise: coding means formultiplexing the additional information generated by the generatingmeans with the multiplexed stream containing the converted video streamand then coding a multiplexed result.

The converting means may convert a video stream picture frame parameter.

The conversion by the converting means may include at least a process ofdecoding the video stream separated by the separating means and aprocess of encoding the decoded video stream.

The additional information generated by the generating means may containat least one of an original picture frame information and an originalscreen aspect ratio.

The additional information generated by the generating means may containan original video format and a video format after the conversion.

The additional information generated by the generating means may containan original screen aspect ratio and a screen aspect ratio after theconversion.

The additional information generated by the generating means may containat least one of information indicating whether or not a picture frame ofthe video stream has been converted by the converting means, informationabout an original picture frame of the video stream separated by theseparating means, and an original screen aspect ratio.

In carrying out the invention and according to a second aspect thereof,there is provided a first image coding method comprising the steps of:inputting a multiplexed stream containing multimedia coding data;separating a video stream from the multiplexed stream input in theinputting step; performing a predetermined converting process on thevideo stream separated in the separating step; generating additionalinformation indicating that a mismatch will occur when the convertedvideo stream is displayed on the basis of the multimedia coding data;and outputting the converted video stream, the multimedia coding data,and the additional information.

The first image coding method may further comprise the step of: codingthe additional information generated in the generating step as dataseparate from the multiplexed stream containing the converted videostream.

The first image coding method may further comprise: a coding step ofmultiplexing the additional information generated in the generating stepwith the multiplexed stream containing the converted video stream andthen coding a multiplexed result.

A video stream picture frame parameter may be converted in theconverting step.

The conversion in the converting step may include at least a process ofdecoding the video stream separated in the separating step and a processof encoding the decoded video stream.

The additional information generated in the generating step may containat least one of original picture frame information and an originalscreen aspect ratio.

The additional information generated in the generating step may containan original video format and a video format after the conversion.

The additional information generated in the generating step may containan original screen aspect ratio and a screen aspect ratio after theconversion.

The additional information generated in the generating step may containat least one of information indicating whether or not a picture frame ofthe video stream has been converted in the converting step, informationabout an original picture frame of the video stream separated in theseparating step, and an original screen aspect ratio.

In carrying out the invention and according to a third aspect thereof,there is provided a first method for recording a computer-readableprogram on a first recording medium, the method comprising: inputting amultiplexed stream containing multimedia coding data; separating a videostream from the multiplexed stream input; performing a predeterminedconverting process on the video stream separated from the multiplexedstream to form a converted video stream; generating additionalinformation indicating that a mismatch will occur when the convertedvideo stream is displayed on the basis of the multimedia coding data;and outputting the converted video stream, the multimedia coding data,and the additional information.

In carrying out the invention and according to a fourth aspect thereof,there is provided a second computer-readable medium having data storedthereon for processing by an image decoding apparatus so that multimediacontent is presented in an intended manner by a display. Thecomputer-readable medium having a data recording area data representinga converted video stream, at least another stream, and multimedia codingdata for controlling display of the video stream and the at leastanother stream on a common display device, the converted video streambeing generated by performing a predetermined conversion process on anoriginal video stream, the predetermined conversion process beingcontrolled by additional information such that a display mismatchbetween the converted video stream and the at least another stream isavoided, the additional information being based on the multimedia codingdata.

The additional information may be coded and recorded as data differentfrom a multiplexed stream containing the converted video stream.

The additional information may be coded and recorded as multiplexed witha multiplexed stream containing the converted video stream.

The video stream may be converted in its picture frame parameter.

The video stream may be decoded and then encoded.

The additional information may contain at least one of original pictureframe information and an original screen aspect ratio.

The additional information may contain information about an originalvideo format and information about a video format after the conversion.

The additional information may contain information about an originalscreen aspect ratio and information about a screen aspect ratio afterthe conversion.

The additional information may contain at least one of informationindicating whether a picture frame of the video stream has beenconverted, information about an original picture frame of the videostream, and information about an original screen aspect ratio.

In carrying out the invention and according to a fifth aspect thereof,there is provided a first image decoding apparatus comprising: inputtingmeans for inputting a multiplexed stream containing multimedia codingdata; separating means for separating a video stream from themultiplexed stream input by the inputting means; decoding means fordecoding the video stream separated from said multiplexed stream by theseparating means; and processing means for performing a predeterminedconversion process on the decoded video stream in accordance withadditional information indicating an occurrence of a mismatch whendisplaying the decoded video stream on the basis of the multimediadecoding data.

The first image decoding apparatus may further comprise: acquiring meansfor acquiring the additional information from data different than themultiplexed stream.

The first image decoding apparatus may further comprise: acquiring meansfor acquiring the additional information from a multiplexed streamcontaining the additional information.

The processing means may convert a picture frame parameter of the videostream.

The conversion by the processing means may include at least a process ofdecoding the video stream separated by the separating means from saidmultiplexed stream and a process of encoding the decoded video stream.

The additional information may contain at least one of information aboutan original picture frame and information about an original screenaspect ratio.

The additional information may contain an original video format andinformation about a video format after the conversion.

The additional information may contain information about an originalscreen aspect ratio and information about a screen aspect ratio afterthe conversion.

The additional information may contain at least one of informationindicating whether a picture frame of the video stream has beenconverted by the converting means, information about an original pictureframe of the video stream separated by the separating means, andinformation about an original screen aspect ratio.

In carrying out the invention and according to a sixth aspect thereof,there is provided a first image decoding method comprising the steps of:inputting a multiplexed stream containing multimedia coding data;separating a video stream from the multiplexed stream input in theinputting step; decoding the video stream separated in the separatingstep; and performing a predetermined conversion process on the decodedvideo stream in accordance with additional information indicating anoccurrence of a mismatch when displaying the decoded video stream on thebasis of the multimedia decoding data.

The first image decoding method may further comprise the step of:acquiring the additional information from data different than themultiplexed stream.

The first image decoding method may further comprise the step of:acquiring the additional information from a multiplexed streamcontaining the additional information.

A picture frame parameter of the video stream may be converted in theprocessing step.

The conversion in the processing step may include at least a process ofdecoding the video stream separated from the multiplexed stream in theseparating step and a process of encoding the decoded video stream.

The additional information may contain at least one of information aboutan original picture frame and information about an original screenaspect ratio.

The additional information may contain an original video format andinformation about a video format after the conversion.

The additional information may contain information about an originalscreen aspect ratio and information about a screen aspect ratio afterthe conversion.

The additional information may contain at least one of informationindicating whether a picture frame of the video stream has beenconverted in the converting step, information about an original pictureframe of the video stream separated in the separating step, andinformation about an original screen aspect ratio.

In carrying out the invention and according to a seventh aspect thereof,there is provided a second method for recording a computer-readableprogram on a third recording medium, the method comprising the steps of:inputting a multiplexed stream containing multimedia coding data;separating a video stream from the multiplexed stream input in theinputting step; decoding the video stream separated from the multiplexedstream in the separating step; and performing a predetermined conversionprocess on the decoded video stream in accordance with additionalinformation indicating occurrence of a mismatch when displaying thedecoded video stream on the basis of the multimedia decoding data.

In carrying out the invention and according to an eighth aspect thereof,there is provided a second image coding apparatus comprising: inputtingmeans for inputting a multiplexed stream; separating means forseparating a video stream from the multiplexed stream input by theinputting means; determining means for determining whether multimediacoding data is contained in the multiplexed stream input by theinputting means; generating means for generating coding controlinformation based on the presence of said multimedia coding data forinstructing that a display format of the video stream separated by theseparating means is not changed; converting means for performing apredetermined conversion process on the video stream separated by theseparating means on the basis of the coding control informationgenerated by the generating means; and multiplexing means for generatinga multiplexed stream that contains the video stream converted by theconverting means.

The generating means may provide an instruction not to change any of apicture frame, a video format, and an aspect ratio.

In carrying out the invention and according to a ninth aspect thereof,there is provided a second image coding method comprising the steps of:inputting a multiplexed stream; separating a video stream from themultiplexed stream input in the inputting step; determining whethermultimedia coding data is contained in the multiplexed stream input inthe inputting step; generating coding control information based on thepresence of said multimedia coding data in said multiplexed stream forinstructing that a display format of the video stream separated in theseparating step is not changed; performing a predetermined conversionprocess on the video stream separated in the separating step on thebasis of the coding control information generated in the generatingstep; and generating a multiplexed stream that contains the video streamconverted in the converting step.

Coding control information for providing an instruction not to changeany of a picture frame, a video format, and an aspect ratio may begenerated in the generating step.

In carrying out the invention and according to a tenth aspect thereof,there is provided a third method for recording a computer-readableprogram on a fourth recording medium, the method comprising the stepsof: inputting a multiplexed stream; separating a video stream from themultiplexed stream input in the inputting step; determining whethermultimedia coding data is contained in the multiplexed stream input inthe inputting step; generating coding control information based on thepresence of the multimedia coding in the multiplexed stream forinstructing that a display format of the video stream separated in theseparating step is not changed; performing a predetermined conversionprocess on the video stream separated in the separating step on thebasis of the coding control information generated in the generatingstep; and generating a multiplexed stream that contains the video streamconverted in the converting step.

In carrying out the invention and according to an eleventh aspectthereof, there is provided a fifth recording medium comprising a datarecording area, wherein coding control information is recorded forinstructing that a display format is not changed for a video stream anda multiplexed stream containing a video stream on which a predeterminedconversion process has been performed on the basis of the coding controlinformation.

As described and according to the first image coding apparatus andmethod and the program stored in the first recording medium, a videosteam is separated from a multiplexed stream containing multimediacoding data, and a predetermined conversion process is performed on theseparated video stream, and additional information indicating that amismatch will occur when displaying the converted video stream on thebasis of the multimedia coding data.

The second recording medium stores a video stream converted by apredetermined conversion process, the multimedia coding data, and theadditional information indicating that a mismatch will occur whendisplaying the converted video stream on the basis of theabove-mentioned multimedia coding data.

As described and according to the image decoding apparatus and methodand the program stored in the third recording medium, when a videostream is separated from an input multiplexed stream, the separatedvideo stream is decoded, and the decoded video stream is displayed onthe basis of multimedia coding data, a mismatch occurs. On the basis ofthe additional information about this mismatch occurrence, apredetermined conversion process is performed on the decoded videostream. This novel configuration prevents the mismatch from occurringbetween the video stream and the multimedia coding data.

As described and according to the second image coding apparatus andmethod and the program stored in the fourth recording medium, a videostream is separated from an input multiplexed stream, the inputmultiplexed stream is checked whether or not multimedia coding data iscontained and, if the multimedia coding data is contained, codingcontrol information for giving an instruction not to change the displayformat of the separated video stream is generated, and a predeterminedconversion process is performed on the separated video stream on thebasis of the generated coding control information.

The fifth recording medium also stores the above-mentioned codingcontrol information giving instruction not to change the display formatof a video stream and a multiplexed stream containing the video streamon which a predetermined conversion process has been performed on thebasis of the coding control information.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other objects of the invention will be seen by reference tothe description, taken in connection with the accompanying drawings, inwhich:

FIGS. 1A and 1B are schematic diagrams illustrating a display screen tobe shown on the basis of multimedia coding information;

FIGS. 2A and 2B are schematic diagrams illustrating a mismatch whichtakes place when a video stream is re-encoded;

FIG. 3 is a block diagram illustrating a recording apparatus practicedas one embodiment of the present invention;

FIGS. 4A and 4B illustrate the operation of a multiplexer shown in FIG.3;

FIGS. 5A, 5B and 5C illustrate the processing by an arrival timestampadding block;

FIG. 6 illustrates multimedia display sub-information;

FIG. 7 illustrates an example of ProgramInfo( ) syntax;

FIG. 8 illustrates an example of StreamCodingInfo( ) syntax;

FIG. 9 illustrates the meaning of stream_coding type;

FIG. 10 illustrates the meaning of video_format;

FIG. 11 illustrates the meaning of frame_rate;

FIG. 12 illustrates the meaning of display_aspect_ratio;

FIG. 13 is a flowchart describing the processing of coding AV stream andmultimedia display sub-information;

FIG. 14 is a flowchart describing the coding processing to be executedfor restricting the re-encoding of a multiplexed stream video includingmultimedia coding data;

FIG. 15 illustrates an example of an input transport stream;

FIG. 16 illustrates an example of a transport stream after there-encoding of the video stream shown in FIG. 15;

FIG. 17 is a flowchart describing a recording rate control process by arecording apparatus shown in FIG. 3;

FIG. 18 is a flowchart describing another recording rate control processby the recording apparatus shown in FIG. 3;

FIG. 19 illustrates another example of a transport stream resulting fromthe re-encoding of the video stream;

FIG. 20 illustrates another example of the input transport stream;

FIG. 21 is a block diagram illustrating a configuration of a reproducingapparatus practiced as one embodiment of the present invention;

FIGS. 22A and 22B illustrate a display screen to be shown whenmultimedia display sub-information is added;

FIG. 23 is a block diagram illustrating another configuration of therecording apparatus practiced as one embodiment of the presentinvention;

FIG. 24 is a flowchart describing the processing of reproducing an AVstream which uses multimedia display sub-information;

FIG. 25 is a block diagram illustrating another configuration of thereproducing apparatus practiced as one embodiment of the presentinvention; and

FIG. 26 illustrates recording media.

DETAILED DESCRIPTION

This invention will be described in further detail by way of examplewith reference to the accompanying drawings. Now, referring to FIG. 3,there is shown a block diagram illustrating an exemplary configurationof a recording apparatus 1 practiced as one embodiment of the invention.A transport stream received at an antenna, not shown, is input in aselector 10. A program number (a channel number) specified by the useris also input from a terminal 11 to the selector 10. Referring to thereceived program number, the selector 10 extracts the specified programfrom the received transport stream and outputs a partial transportstream. The partial transport stream is input in a demultiplexer 12 andan analyzing block 13.

The partial transport stream input in the demultiplexer 12 is separatedinto a video stream and other streams (audio, still picture, charactergraphics, and multimedia coding data for example). The video stream thusobtained is output to a decoder 14. The other streams are output to amultiplexer 16. In addition to the transport packets other than video,the demultiplexer 12 outputs the output timing information in the inputtransport stream of these transport packets to the multiplexer 16.

The decoder 14 applies a predetermined decoding scheme, for example,MPEG2 to the input video stream and outputs the decoded video data to anencoder 15. Also, the decoder 14 outputs the stream information aboutthe video stream obtained at decoding to a coding controller 18.

On the other hand, the analyzing block 13 analyzes the input transportstream to obtain the stream information about the non-video streams, forexample, a bit rate, and outputs it to the coding controller 18. Thestream information about the non-video streams output from the analyzingblock 13, the video stream information output from decoder 14, and astream recording bit rate output from a terminal 19 are input in thecoding controller 18. From these data, the coding controller 18 sets thevideo data coding conditions (coding control information) to be executedby the encoder 15 and outputs these coding conditions to the encoder 15and a coding block 20.

The coding controller 18 uses, as a bit rate to be allocated to thevideo data encoding, a value obtained by subtracting a total value (thedata input from the analyzing block 13) of the bit rates of thenon-video streams from a stream recording bit rate (the data input, viathe terminal 19, from a controller, not shown, for controlling theoperation of the recording apparatus 1, for example). The codingcontroller 18 sets coding control information such as bit rate andpicture frame such that an optimum picture quality can be achieved withthe bit rate thus obtained and outputs this coding control informationto the encoder 15 and the coding block 20. The details of the codingcontrol information will be described later with reference to FIGS. 15through 20.

When a stream is recorded to a recording medium with a fixed rate, thisstream recording bit rate becomes the fixed rate; if a stream isrecorded with a variable bit rate, this stream recording bit rate is amean bit rate per predetermined time. However, the maximum value of thevariable bit rate in this case needs to be lower than the maximumrecording bit rate ensured by the recording medium concerned.

The encoder 15 encodes (on the basis of MPEG2, for example) the videodata output from the decoder 14 on the basis of the coding controlinformation output from the coding controller 18 and outputs theresultant video data to the multiplexer 16. The video stream from theencoder 15, the transport stream packets other than video from thedemultiplexer 12, and the information about the occurrence timing of thetransport stream packets other than video are input in the multiplexer16. On the basis of the input occurrence timing information, themultiplexer 16 multiplexes the video stream with the transport streampackets, other than video, and outputs the result to the arrivaltimestamp adding block 17 as a transport stream.

FIGS. 4A and 4B schematically illustrate the above-mentioned processingto be executed by the multiplexer 16. FIG. 4A shows the timing of theinput transport stream packets. In these figures, the cross-hatchedportions indicate the video packets while the white portions indicatethe stream packets other than video. As shown in FIG. 4A, the inputtransport stream packets are continuous; however, the data volume of thevideo data is reduced by the re-encoding of video data by the encoder15. Consequently, the number of video packets is reduced.

As shown in FIG. 4B, the multiplexer 16 does not change the timing ofthe stream packets other than video but causes only the timing of thevideo packets to be different from the original state (shown in FIG.4A).

As shown in FIGS. 5A, 5B and 5C, the arrival timestamp adding block 17adds a header (TP_extra_header) including an arrival timestamp to eachof the packets (FIG. 5A) of the input transport stream to generate asource packet (FIG. 5B), arranges the generated source packetscontinuously (FIG. 5C), and outputs them to a writing block 21. Thearrival timestamp is information indicative of the timing with which thetransport stream packets occur in a transport stream. The writing block21 takes the input source packet stream consisting of continuous sourcepackets and records the file to a recording medium 22. It should benoted that the recording medium 22 may be any type of recording medium.

The information output from the coding block 20 is also input in thewriting block 21. On the basis of the video coding information from thecoding controller 18, the coding block 20 generates multimedia displaysub-information and outputs the same to the writing block 21. Themultimedia display sub-information to be output to the writing block 21is information for keeping the video display position and display sizeunchanged on multimedia plane from those of the image (the image whichwould be displayed without re-encoding) intended by the sending sideeven if the picture frame size has changed by transcoding (decoding bythe decoder 14 and then encoding by the encoder 15) a video stream. Thisinformation also is used at the time of reproduction in combination withmultimedia coding data.

The following describes the multimedia display sub-information morespecifically. As shown in FIG. 6, the multimedia display sub-informationconsists of three flags of a mismatch flag (mismatch_MMinfo_flag), are-encoded flag (Re_encoded_flag), and a frame size change flag(changed_frame_size_flag), data associated with two sizes indicative ofan original horizontal size (original_horizontal_size) and an originalvertical size (original_vertical_size), and an original screen aspectratio (original_display_aspect_ratio).

The mismatch flag indicates whether there exists a mismatch in therelationship between video and multimedia coding data. The re-encodedflag indicates whether the video has been re-encoded at the time ofrecording. The frame size change flag indicates whether the pictureframe of video has been changed by re-encoding, for example. Theoriginal horizontal size indicates the horizontal size of a pictureframe before re-encoding. The original vertical size indicates thevertical size of a picture frame before re-encoding. The original screenaspect ratio indicates the aspect ratio of a frame screen beforere-encoding.

It should be noted that the above-mentioned multimedia displaysub-information is illustrative only. Therefore, information other thanthat shown in FIG. 6 may be included in, or part of the informationshown in FIG. 6 may be excluded from, the multimedia displaysub-information.

The following describes another example of the multimedia displaysub-information. In the following example, the multimedia displaysub-information is stored in a ProgramInfo( ) syntax shown in FIG. 7.The following describes the fields associated with the present inventionin the ProgramInfo( ) syntax.

“length” indicates the number of bytes between the byte just after thelength field and the last byte of ProgramInfo( ) inclusive.

“num_of_program_sequences” indicates the number of program sequences inthe an AV stream file. A source packet sequence with which the programcontents specified by this format in the AV stream file are constant isreferred to as a program sequence.

“SPN_program_sequences_start” indicates an address at which the programsequence starts in the AV stream file. “SPN_program_sequences_start” isof a size in unit of source packet number and counted from the initialvalue 0 starting with the first packet of the AV stream file.

“program_map_PID” is value of the PID of a transport packet having PMT(Program Map Table) applicable to that program sequence.

“num_of_streams_in_ps” indicates the number of elementary streamsdefined in that program sequence.

“stream_PID” indicates the value of the PID for the elementary streamdefined in the PMT which is referenced by the program map PID of thatprogram sequence.

“StreamCodingInfo( )” indicates the information about the elementarystream indicated by the above-mentioned stream PID.

FIG. 8 shows the syntax of StreamCodingInfo( ). “length” indicates thenumber of bytes between the byte just after this length field and thelast byte of StreamCodingInfo( ) inclusive.

“stream_coding_type” indicates the coding type of the elementary streamindicated by the stream PID for this StreamCodingInfo( ). The meaningsof the individual types are shown in FIG. 9.

If the value of stream coding type is 0x02, it indicates that theelementary stream indicated by the stream PID is a video stream.

If the value of stream coding type is 0x0A, 0x0B, or 0x0D, it indicatesthat the elementary stream indicated by the stream PID is multimediacoding data.

If the value of stream coding type is 0x06, it indicates that theelementary stream indicated by the stream PID is subtitles or teletext.

“video_format” indicates the video format of a video stream indicated bythe stream PID for this StreamCodingInfo( ). The meanings of theindividual video formats are shown in FIG. 10.

In FIG. 10, 480i indicates video display of NTSC standard TV (interlaceframe of 720 pixels×480 lines). 576i indicates video display of PALstandard TV (interlace frame of 720 pixels×576 lines). 480p indicatesvideo display of progressive frame of 720 pixels×480 lines. 1080iindicates video display of interlace frame of 1920 pixels×1080 lines.720p indicates video display of progressive frame of 1230 pixels×720lines.

“frame_rate” indicates the frame rate of a video stream indicated by thestream PID for this StreamCodingInfo( ). The meanings of the individualframe rates are shown in FIG. 11.

“display_aspect_ratio” indicates the display aspect ratio of a videostream indicated by the stream PID for this StreamCodingIndo( ). Themeaning of the individual display aspect ratios are shown in FIG. 12.

“original_video_format_flag” indicates whether there exists originalvideo format and original display aspect ratio in this StreamCodingInfo().

“original_video_format” indicates a video format before a video streamindicated by the stream PID for this StreamCodingInfo( ) is coded. Themeanings of the individual original video formats are the same as shownin FIG. 10.

“original_display_aspect_ratio” is the display aspect ratio before avideo stream indicated by the stream PID for this StreamCodingInfo( ) iscoded. The meanings of the individual aspect ratios are the same asshown in FIG. 12.

It is assumed that, in transcoding a transport stream with a multimediadata stream (BML stream or subtitles) multiplexed along with a videostream, the re-encoding of the video stream changes its video format(for example, from 1080i to 480i), while the multimedia data streamretains its original stream contents. In this case, a mismatch ininformation may occur between a new video stream and the multimedia datastream. For example, although the parameters associated with the displayof the multimedia data stream are determined on the supposition of thevideo format of the original video stream, the video format may bechanged by the re-encoding of the video stream.

The video format of the original video stream is indicated by the videoformat and the display aspect ratio. The video format of the re-encodedvideo stream is indicated by the original video format and the originaldisplay aspect ratio.

If a mismatch exists between the values of the video format and theoriginal video format and/or between the display aspect ratio and theoriginal display aspect ratio, it indicates that a video format changehas been caused by the video re-encoding at the time of recording.

If the stream PID in which the stream coding type indicates multimediacoding data and subtitles are included in ProgramInfo( ), it indicatesthat the multimedia data is multiplexed in an AV stream file (atransport stream).

If ProgramInfo( ) indicates that a video format change has been causedby the re-encoding of video at the time of recording and multimedia datais multiplexed in the AV stream file, then it is determined that amismatch exists in display between the video stream (re-encoded) and themultimedia data (the original multimedia data) in the AV stream file.

In such a case, the information about the original video stream, namelythe original video format and the original display aspect ratio, becomeseffective. The reproducing apparatus generates a display screen from theabove-mentioned new video stream and multimedia data stream as follows.

The video stream is up-sampled to a video format indicated by theoriginal video format and the original display aspect ratio.

The up-sampled image and the multimedia data stream are synthesized toform a correct display screen.

The multimedia display sub-information generated by the coding block 20is recorded by the writing block 21 to the recording medium 22 butstored as a file which is different from the source packet stream fileoutput from the arrival timestamp adding block 17. If the multimediadisplay sub-information is recorded by the writing block 21 to therecording medium 22 as a file different from the source packet streamfile, the filed multimedia display sub-information is output from thecoding block 20.

FIG. 13 is a flowchart describing the processing of coding an AV streamand multimedia display sub-information.

In step 50, a multiplexed stream including multimedia coding data isinput in the recording apparatus 1.

In step 51, the demultiplexer 12 separates the video stream from themultiplexed stream.

In step 52, the encoder 15 re-encodes the video stream decoded by thedecoder 14.

In step 53, the multiplexer 16 multiplexes the above-mentioned videostream and multimedia coding data to generate a multiplexed stream.

In step 54, the coding block 20 generates multimedia displaysub-information.

In the above description, the coding controller 18 generates the codingcontrol information including bit rate and picture frame on the basis ofthe input data. The coding controller 18 may generate the followinginformation as alternative coding control information. Namely, if theinput transport stream is found to include multimedia coding data by theanalyzing block 13, then the coding controller 18 may generate codingcontrol information when encoding is executed by the encoder 15 forinstructing the encoder 15 to execute the re-encoding with a pictureframe (the picture frame before re-encoding) of the same size as that ofthe picture frame of the original video, and output the generated codingcontrol information to the encoder 15.

When the above-mentioned method is used, the encoder 15 re-encodes thevideo data supplied from the decoder 14 with the same value as that ofthe picture frame of the original video stream on the basis of the inputcoding control information. If such coding control information isgenerated and the re-encoding is executed on the basis of the codingcontrol information, no picture frame change is caused by there-encoding, thereby preventing a mismatch from occurring in therelationship between the video stream obtained by re-encoding and themultimedia coding data.

Still alternatively, the following information may be generated as thecoding control information generated by the coding controller 18.Namely, if the input transport stream is found to include multimediacoding data by the analyzing block 13, then the coding controller 18 maygenerate coding control information when encoding is executed by theencoder 15 for instructing the encoder 15 to execute the re-encodingunder the same conditions as the video format (shown in FIG. 10) andscreen aspect ratio (shown in FIG. 12) of the original video, and outputthe coding control information to the encoder 15.

When the above-mentioned method is used, the encoder 15 re-encodes thevideo supplied from the decoder 14 under the same conditions as thevideo format (shown in FIG. 10) and screen aspect ratio (shown in FIG.12) of the original video on the basis of the input coding controlinformation. If such coding control information is generated and there-encoding is executed on the basis of the coding control information,no video format and no screen aspect ratio change is caused by there-encoding, thereby preventing a mismatch from occurring in therelationship between the video stream obtained by re-encoding and themultimedia coding data.

FIG. 14 is a flowchart describing the coding for restricting there-encoding of the video of a multiplexed stream including multimediacoding data.

In step 70, a multiplexed stream is input in the recording apparatus 1.

In step 71, the demultiplexer 12 separates the video stream from themultiplexed stream.

In step 72, the analyzing block 13 checks if the multimedia coding datais included in the video stream. If the multimedia coding data isincluded, the analyzing block 13 sends the coding control information tothe encoder 15 instructing the same to re-encode the video streamwithout changing the display format. On the basis of the suppliedcontrol information, the encoder 15 re-encodes the video stream.

In step 73, the multiplexer 16 generates a multiplexed stream includingthe above-mentioned video stream.

With reference to FIGS. 15 through 20, the following describes oneexample of control to be executed on the basis of the coding controlinformation.

It is assumed here that a transport stream to be input to the selector10 has a constant bit rate R_(I) as shown in FIG. 15, for example. Thevideo stream and the non-video streams are coded by variable bit rates.In the example shown in FIG. 15, in unit time (for example, GOP) A, thebit rate of the video stream is R_(VA) and the bit rate of non-videostreams is R_(OA). In unit time B, the bit rate of the video stream isR_(VB) and the bit rate of non-video streams is R_(OB). In unit time C,the bit rate of the video stream is R_(VC) and the bit rate of non-videostreams is R_(OC).

If the transport stream as shown in FIG. 15 is re-encoded to output thetransport stream having fixed bit rate S (S<R_(I)) as shown in FIG. 16from the multiplexer 16, the coding controller 18 executes theprocessing described by the flowchart shown in FIG. 17.

First, in step S1, the coding controller 18 sets the bit rate to S(recording rate) of a transport stream to be output from the multiplexer16 on the basis of a control signal input from a controller, not shown,via the terminal 19. Next, in step S2, the coding controller 18determines non-video streams to be recorded and computes a maximum totalvalue D of the bit rates of the determined streams.

The maximum value D is determined from the stream specification of theinput transport stream. For example, if two audio streams are to berecorded in addition to the video stream, the maximum value D is 384×2Kbps since the maximum value of the bit rate of one audio stream is 384Kbps according to the Japanese digital BS broadcast streamspecification.

In step S3, the coding controller 18 uses value C obtained bysubtracting the maximum value D computed in step S2 from the recordingbit rate set in step S1 (C=S−D), as a bit rate to be allocated to there-encoding of the video data. In step S4, the coding controller 18analyzes the coding information such as the video stream bit rate andpicture frame from the video stream information output from the decoder14.

In step S5, the coding controller 18 determines, on the basis of thevalue C computed in step S3 and the video stream coding informationanalyzed in step S4, a video coding parameter (video coding controlinformation) such that an optimum picture quality is achieved.

For example, in the example shown in FIG. 16, value S is ½ of valueR_(I). In the present example, the bit rate of steams other than videois the maximum value D, which is used without change as the bit rate ofnon-video steams in a multiplexed stream after re-encoding.

Then, video coding parameters are determined such that an optimumpicture quality can be achieved within the range of (S−D). If thepicture frame is controlled, the horizontal direction of a picture frameof 720×480 pixels, for example, is sampled by ½ into 360×480 pixels. Thedetermined, coding parameters (bit rate and picture angle) are suppliedto the encoder 15 as video coding control information.

In step S6, on the basis of the video coding control informationsupplied from the coding controller 18, the encoder 15 re-encodes thevideo data of unit time (in this example, unit time A) to be processednow. In the example shown in FIG. 16, the actual bit rate R_(OA) issmaller than the maximum value D in unit time A; however, since themaximum value D is fixed, the video allocated bit rate becomes (S−D). Awasted portion Rsa which cannot be used for video coding occurs becausethe maximum value D is fixed. The wasted portion is filled with stuffingbits.

In step S7, the coding controller 18 determines whether there remainsany stream to be re-encoded. If any streams remain to be re-encoded, theprocedure returns to step S4 to repeat the above-mentioned processes.

If, in step S7, no more streams remain to be re-encoded, this processingcomes to an end.

Thus, in the example shown in FIG. 16, in unit time B, the bit rate ofnon-video streams also is D and the video stream allocated bit rate isS−D because it is fixed. Stuffing bits are inserted in value R_(sb)(R_(sb)=S−(S−D)−R_(OB)=D−R_(OB)).

In unit time C, too, the bit rate of non-video streams is D and thevideo stream allocated bit rate is S−D. It should be noted that, in unittime C,

D=R_(OC), so that no stuffing bits exist.

Thus, in the example shown in FIG. 16, the video stream is coded with afixed bit rate.

FIG. 18 is a flowchart describing a processing example in which thevideo re-encoding allocated bit rate is variable. First, in step S21,the coding controller 18 sets recording rate S on the basis of theinformation supplied via the terminal 19. Next, in step S22, the codingcontroller 18 analyzes the coding information of the video stream on thebasis of the video stream information supplied from the decoder 14. Theprocesses of steps S21 and S22 are the same as those of steps S1 and S4of FIG. 17.

In step S23, the coding controller 18 computes, from the output of theanalyzing block 13, the total bit rate B in each unit time of non-videostreams.

In step S24, the coding controller 18 uses, as the video re-encodingallocated bit rate, value C (C=S−B) obtained by subtracting value Bobtained in step S23 from value S obtained in S1.

In step S25, the coding controller 18 determines, on the basis of valueC obtained in step S24 and a result of analysis of the video streamcoding information obtained in step S22, video coding parameters suchthat an optimum picture quality is obtained. The determined codingparameters are output to the encoder 15.

In step S26, the encoder 15 re-encodes the video data of the currentunit time on the basis of the coding parameters determined in step S25.Consequently, as shown in FIG. 19, for example, after allocation ofR_(oa) (=R_(OA)) as the bit rate in unit time of non-video streams, thebit rate of the video stream is set to bit rate R_(va) specified by(S−R_(oa)).

In step S27, the coding controller 18 determines whether any streamsremain to be processed. If any streams remain to be processed, theprocedure returns to step S22 to repeat the above-mentioned processes.If no more streams remain to be processed, this processing comes to anend.

Thus, in unit time B, after allocation of bit rate R_(ob) (=S−R_(OB)) ofnon-video streams, the remaining R_(vb) (=S−R_(ob)) is the bit rate ofthe video stream. In unit time C, the bit rate of the video stream isset to R_(vc) (=S−R_(OC)), except for bit rate Roc of non-video streams.

Thus, in the present processing example, the bit rate of the videostream is variable and, therefore, no stuffing bit is needed or thenumber of stuffing bits can be reduced, thereby coding the video streammore efficiently.

In the above, the input transport stream has a fixed bit rate. Thepresent invention also is applicable to an example in which the bit rateof the input transport stream is variable as shown in FIG. 20.

Consequently, a transport stream of longer content can be recorded tothe recording medium 22 at a lower bit rate as required.

In addition, the above-mentioned novel embodiment prevents the qualitiesof audio data, still picture and character graphics data, multimediacoding data, and other non-video data from being conspicuouslydeteriorated. The non-video data is basically smaller in data volumethan video data, so that reducing the bit rate of the non-video data inthe same ratio as the bit rate of video data makes the effects on thenon-video data relatively greater than those on video data. The novelembodiment can prevent these effects from being caused.

The following describes the reproduction of a source packet stream filerecorded on the recording medium 22. Referring to FIG. 21, there isshown a block diagram illustrating the configuration of a reproducingapparatus practiced as one embodiment of the invention. A source packetstream file recorded on the recording medium 22 is read by a readingblock 31. The reading block 31 also reads multimedia displaysub-information recorded on the recording medium 22 as a file separatefrom the source packet stream file.

The source packet stream read by the reading block 31 is output to aarrival timestamp separating block 32 and the multimedia displaysub-information is output to a synthesizing block 36. The arrivaltimestamp separating block 32 incorporates a reference clock. Thearrival time stamp separating block 32 compares the reference clock withthe value of the arrival timestamp added to the source packet of theinput source packet stream and, when a match is found, removes thearrival timestamp from the source packet having the matching arrivaltimestamp, outputting the resultant packet to a demultiplexer 33 as atransport stream packet.

The demultiplexer 33 separates the input transport stream into avideo/audio stream and data streams such as multimedia coding data,character graphics, text, and still picture. Of these separated data,the video/audio stream is output to an AV decoder 34, the multimediacoding data is output to the synthesizing block 36, and the data streamsuch as character graphics, text, and still picture is output to acharacter graphics/still picture decoder 35.

The AV decoder 34 separates the input video/audio stream into video dataand audio data, decodes each data, and outputs the decoded audio data toan audio reproducing device, not shown, and the decoded video data tothe synthesizing block 36. The character graphics/still picture decoder35 decodes the input data stream, such as character graphics, text, andstill picture, and outputs the decoded character graphics data, textdata, and still picture data to the synthesizing block 36.

In the synthesizing block 36, the video data from the AV decoder 34, themultimedia coding data from the demultiplexer 33, the data from thecharacter graphics/still picture decoder 35, and the multimedia displaysub-information from the reading block 31 are input. Checking themismatch flag (FIG. 6) of the input multimedia display sub-information,the synthesizing block 36 determines whether a mismatch exists in therelationship between the input video signal and the multimedia codingdata.

If a mismatch exists between the value of video format and the value oforiginal video format shown in FIG. 8 and/or a mismatch exists betweenthe value of display aspect ratio and the original display aspect ratio,the synthesizing block 36 determines that a video format change has beencaused by the video re-encoding at the time of recording, detecting amismatch in the relationship between the input video signal and themultimedia encoding data. If no mismatch exists between the value ofvideo format and the value of original video format and no mismatchexists between the value of display aspect radio and the value oforiginal display aspect ratio, the synthesizing block 36 determines thatno mismatch exists in the relationship between the input video signaland the multimedia coding data.

If a mismatch is found in the relationship between the input videosignal and the multimedia coding data, the synthesizing block 36 furtherreferences the original horizontal size and vertical size of themultimedia display sub-information or references the original videoformat and the original display aspect ratio. Then, the synthesizingblock 36 scale-converts the input video signal so that it can bedisplayed in a frame of the referenced size. On the basis of themultimedia coding data, the synthesizing block 36 outputs the videosignal with the scale-converted video signal and the data, such ascharacter graphics synthesized on a multimedia plane, to a televisionreceiver, not shown, which serves as a display device.

On the other hand, if no mismatch is found in the relationship betweenthe input video signal and the multimedia coding data, the synthesizingblock 36 synthesizes the input video signal with other data on amultimedia plane without scale conversion and outputs the synthesizeddata.

Thus, recording the multimedia display sub-information and using it atthe time of reproduction allow the receiving side to display a screen asintended on the sending side. Referring to FIG. 22, if the re-encodingon the sending side (recording side) results in a smaller video pictureframe than the original, the size reduction is recorded as multimediadisplay sub-information, which is referenced at the time ofreproduction. Consequently, because there exists no mismatch betweenvideo data and other data, the receiving side (the reproduction side)can display the same screen as the original.

FIG. 24 is a flowchart describing AV stream reproduction processingwhich uses multimedia display sub-information.

In step 60, a multiplexed stream including multimedia coding data isread from a recording medium and input in a reproduction device.

In step 61, multimedia display sub-information is input. Thisinformation is read from the recording medium in the case of thereproducing device shown in FIG. 21; in the case of a reproducing deviceshown in FIG. 25, this information is separated from the multiplexedstream.

In step 62, a video stream is separated from the multiplexed stream.

In step 63, the video stream is decoded.

In step S64, if a mismatch exists between the video data and themultimedia coding data, the synthesizing block 36 scale-converts thevideo data on the basis of the multimedia display sub-information.

In step 65, the synthesizing block 36 synthesizes the processed imageand the multimedia data to generate a display image.

As described, the multimedia display sub-information may be recorded onthe recording medium 22 as a file which is different from the sourcepacket stream file containing character graphics data and video signals.Alternatively, the multimedia display sub-information may be embedded ina source packet stream file and then recorded on the recording medium22. FIG. 23 shows the configuration of the recording apparatus 1 inwhich the multimedia display sub-information is embedded in a sourcepacket stream file.

In comparison between the configuration of the recording apparatus 1shown in FIG. 23 and the configuration shown in FIG. 3, the formeroutputs the multimedia display sub-information output from the codingblock 20 and supplies this output to the multiplexer 16. The multiplexer16 then generates a transport packet of the input multimedia displaysub-information and embeds it into a source packet stream file,outputting the same to the arrival timestamp adding block 17. Instead ofembedding the multimedia display sub-information into a source packetstream file as a transport packet, the multimedia displaysub-information may be written to a user data area in an MPEG videostream.

In the present embodiment of the invention, video data may be re-encodedusing other methods than that described above; for example, an inputvideo stream may be converted in the DCT area to convert the codingparameters such as picture frame.

FIG. 25 shows the configuration of the reproducing apparatus 30 in whichthe multimedia display sub-information is embedded in a source packetstream file to be recorded on the recording medium 22. In comparisonbetween the configuration of the reproducing apparatus shown in FIG. 25and the configuration shown in FIG. 21, the former reads only the sourcepacket stream through the reading block 31. The source packet streamread by the reading block 31 is input to the demultiplexer 33 via thearrival timestamp separating block 32.

The demultiplexer 33 extracts the multimedia display sub-informationfrom the input source packet stream file and outputs the extractedinformation to the synthesizing block 36. The further processing is thesame as that of the configuration shown in FIG. 5.

Thus, if the multimedia display sub-information is recorded as embeddedin a source packet stream file, the receiving side can also obtain thevideo picture size and display position intended by the sending side.

In the present embodiment of the invention, a transport stream was usedas an example. The present invention also is applicable to multiplexedstreams such as a program stream.

The above-described sequence of processing operations can be executed byhardware as well as software. In the software approach, the recordingapparatus 1 (and the reproducing apparatus 30) is constituted by apersonal computer as shown in FIG. 26.

Referring to FIG. 26, a CPU (Central Processing Unit) 101 executesvarious processing operations as instructed by programs stored in a ROM(Read Only Memory) 102 or loaded from a storage block 108 into a RAM(Random Access Memory) 103. The RAM 103 also stores, as required, thedata necessary for the CPU 101 to execute various processing operations.

The CPU 101, the ROM 102, and the RAM 103 are interconnected via a bus104. The bus 104 also is connected to an input/output interface 105.

The input/output interface 105 is connected to an input block 106, suchas a keyboard and a mouse, a display device such as a CRT or LCD, anoutput block 107, such as a speaker, a storage block 108 such as harddisk, and a communication block 109 such as modem or terminal adapter.The communication block 109 executes communication processing via anetwork.

The input/output interface 105 also is connected to a drive 110, asrequired, in which a magnetic disc 121, an optical disc 122, amagneto-optical disc 123, or a semiconductor memory 124 is loaded.Computer programs read from these storage media are installed in thestorage block 108 as required.

The execution of a sequence of processing operations by softwarerequires the use of a computer having a dedicated hardware devicestoring beforehand the programs constituting the software or ageneral-purpose computer in which these programs are installed, asrequired, from a recording medium.

The program recording medium for storing computer-readable andexecutable programs may be a package medium which is distributed tousers providing programs and embodied by, the magnetic disk 121(including floppy disk), the optical disc 122 (including CD-ROM (CompactDisc-Read Only Memory) and DVD (Digital Versatile Disc)), themagneto-optical disk 123 (including MD (Mini Disk)), the semiconductormemory 124, a ROM 102 or a hard disk which is preinstalled in a personalcomputer and provided for users and on which the programs are storedtemporarily or permanently as shown in FIG. 26.

It should be noted that the steps describing the programs to be storedon the program storage medium are not only executed in a time-dependentmanner in the order described, but also in parallel or in a discretemanner.

As described, and according to the first image coding apparatus andmethod and the program stored in the first recording medium, a videosteam is separated from a multiplexed stream containing multimediacoding data, a predetermined conversion process is performed on theseparated video stream, and additional information indicative of amismatch occurs when displaying the converted video stream on the basisof the multimedia coding data.

The first recording medium stores the converted video stream, themultimedia coding data, and the additional information indicative that amismatch will occur when displaying the converted video stream on thebasis of the above-mentioned multimedia coding data.

Consequently, in any case, the reproducing side can prevent a mismatchfrom occurring between the video stream and the multimedia coding data.

As described and according to the image decoding apparatus and methodand the program stored in the second recording medium, a mismatch occurswhen a video stream is separated from an input multiplexed stream, theseparated video stream is decoded, and the decoded video stream isdisplayed on the basis of multimedia coding information. On the basis ofthe additional information about this mismatch occurrence, apredetermined conversion process is performed on the decoded videostream. This novel configuration prevents the mismatch from occurringbetween the video stream and the multimedia coding data.

As described and according to the second image coding apparatus andmethod and the program stored in the third recording medium, a videostream is separated from an input multiplexed stream, the inputmultiplexed stream is checked for multimedia coding data and, if themultimedia coding data is found, coding control information for givingan instruction not to change the display format of the separated videostream is generated, and a predetermined conversion process is performedon the separated video stream on the basis of the generated codingcontrol information.

The second recording medium also stores the above-mentioned codingcontrol information giving instruction not to change the display formatof a video stream and a multiplexed stream containing the video streamon which a predetermined conversion process has been performed on thebasis of the coding control information.

Consequently, in any case, the reproduction side can prevent a mismatchfrom occurring between the video stream and the multimedia coding data.

Although the invention herein has been described with reference toparticular embodiments, it is to be understood that these embodimentsare merely illustrative of the principles and applications of thepresent invention. It is therefore to be understood that numerousmodifications may be made to the illustrative embodiments and that otherarrangements may be devised without departing from the spirit and scopeof the present invention as defined by the appended claims.

1. A computer-readable medium having data stored thereon for processingby an image decoding apparatus so that multimedia content is presentedin an intended manner by a display, said computer-readable mediumcomprising: a data recording area containing data representing aconverted video stream, at least another stream, and multimedia codingdata for controlling display of the video stream and the at leastanother stream on a common display device, the converted video streambeing generated by performing a predetermined conversion process on anoriginal video stream, the predetermined conversion process beingcontrolled by additional information such that a display mismatchbetween the converted video stream and the at least another stream isavoided, the additional information being based on the multimedia codingdata.
 2. The computer-readable medium according to claim 1, wherein saiddata recording area stores the additional information which is coded andstored as data different from a multiplexed stream containing theconverted video stream.
 3. The computer-readable medium according toclaim 1, wherein said data recording area stores the additionalinformation which is coded and stored as data multiplexed with amultiplexed stream containing the converted video stream.
 4. Thecomputer-readable medium according to claim 1, wherein the originalvideo stream is converted in its picture frame parameter.
 5. Thecomputer-readable medium according to claim 1, wherein the originalvideo stream is decoded and then encoded.
 6. The computer-readablemedium according to claim 1, wherein the additional information includesat least one of original picture frame information and an originalscreen aspect ratio.
 7. The computer-readable medium according to claim1, wherein the additional information includes information about anoriginal video format.
 8. The computer-readable medium according toclaim 1, wherein the additional information includes information aboutan original screen aspect ratio.
 9. The computer-readable mediumaccording to claim 1, wherein the additional information includes atleast one of information indicating whether a picture frame of saidvideo stream has been converted, information about an original pictureframe of said video stream, and information about an original screenaspect ratio.
 10. The computer-readable medium according to claim 6,wherein the original picture frame information includes a size of thepicture frame.